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DETAILED ACTION 

Applicant's submission filed on 10/17/2005 has been entered. Claims 1,8, 11, 26-29 have 
been amended. Claims 2-4, 6-7, 9-10, 12-29 have been canceled. Claims 1, 5, 8, and 1 1 are 
pending in the present application. 



Response to Arguments 

Applicant's arguments filed October 17, 2005 have been fully considered but they are not 
found persuasive. As addressed below, Xiong, Shum and Teo teach the claim limitations. 

Applicant argues in essence with respect to the base claim 1 and similar claims that Teo 
does not teach automatically selecting and setting the mapping mode. However, Xiong teaches in 
figures 2 and 3 a user interface and a global optimization that automatically provides feedback to 
the computer system when the pair-wise objective function is not desirable for a poor selection 
of the projection viewing plane and the resulting panoramas will have imperfectly aligned 
images that give shadow or ghosting effects due to a poor selection of the projection viewing 
plane and the resulting panoramas will have imperfectly aligned images that give shadow or 
ghosting effects. Xiong further teaches an alternative human interaction being present throughout 
the image synthesis process to provide feedback to the computer system in all the nonlinear 
optimizations to let users monitor the progress of the system and allow them to intervene when 
necessary . Allowing user's intervening in the optimization process does not mean Xiong teaches 
away from automatic selecting and setting of the mapping mode. 
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In fact, Xiong discloses in column 8, lines 18-58 that a panorama is constructed on a 
particular geometry that will best facilitate the rendering of the projection onto a chosen 
viewing plane (a geometric surface) such as cubic, polyhedral, cylindrical and spherical 
geometries. Xiong describes that a user can alternatively intervene to select a viewing plane. 
Xiong at least discloses that upon the user's intervening of selecting a viewing plane, the 
computer program automatically selecting and setting the mapping mode in accordance to the 
parameters set by the user's intervening action. However, without the user's intervening action, 
the non-linear optimization process decides the optimal mapping mode and thus Xiong at least 
suggests automatically setting a mapping surface without a user intervention to determine the 
mapping mode. 

Teo teaches that the viewing software can automatically invert the projection by 
automatically straightening out the portion of the panoramic image being displayed and 

thereby automatically selecting and setting the mapping mode by the viewer software (column 7- 
8). Furthermore, the claim limitation of "automatically selecting and setting the mapping mode" 
does not necessarily imply automatically setting the mapping surface. Teo teaches the image 
processor for automatically transforming the first digital panoramic image into a second digital 
panoramic image corresponding to the projection of the scene onto the second surface geometry 
and therefore automatically selecting and setting the mapping mode of mapping the images to the 
selected surface geometry (column 7). Teo teaches in column 10 the mapping mode of mapping 
onto the planar surfaces such as mapping the panorama images onto a cube and the mapping 
mode of mapping the panorama images onto polyhedral surfaces to form cylindrical panorama 
images. Once the geometry is selected either by a user or by the viewer software , the image 
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processor automatically transforms the first digital panoramic image into a second digital 
panoramic image corresponding to the projection of the scene onto the surface geometry. 
Therefore, the mapping mode is automatically selected and set by the viewer software because 
the steps involved a mode in the mapping or a mode in transforming the first digital panoramic 
image into a second digital panoramic image corresponding to the projection of the scene onto 
the changed/selected geometric surface are automatically performed by the software. Although 
Teo also teaches a user can intervene in selecting a mapping surface, however, the mapping 
mode of mapping the first digital panoramic image into a second digital panoramic image 
corresponding to the projection of the scene onto a selected geometric surface is 
automatically selected and set by the software. It is also noted that each mapping mode of 
transforming the panoramic scene corresponds to a selected geometric surface. 

Although Teo teaches that a user can change a mapping surface ( column 11, lines 1-9 ) by 
adjusting the shape of the mapping surfaces so that a given cylindrical panoramic image can be 
transformed to produce a modified panoramic image, the procedure can be automated by the 
computer software. Teo therefore teaches automatically selecting and setting a mapping mode or 
changing the plane for mapping the panoramic image by the computer through a user interface or 
by the computer software without the user interface. 

Teo further discloses the horizontal placement of image by projecting a scene onto a cube 
so that horizontal lines of the scene appear horizontal in the panoramic image, and vertical 
placement of image by projecting the panoramic image on the vertical cylinder or by rotating the 
polyhedral surface relative to a cylindrical panoramic image and making adjustments to the 
sides and edges of the polyhedral surface so that the placement of images appear vertically 
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on the mapping surface and thus selecting and setting a mapping mode with the mapping 
surface being adjusted without a user intervention. Teo teaches the user interface enabling the 
user to rotate the polyhedral surface relative to a cylindrical panoramic image and making 
adjustments to the sides and edges of the polyhedral surface so that the placement of images 
appear vertically on the mapping surface. 

Shum teaches a plurality of mapping modes such as zooming and panning the screen 
images of the interior scene shown in the panoramic view of Fig. 10A wherein the mapping 
surfaces within panoramas are different (See Figs. 1 1A and 1 IB and column 23-24) and 
changing the type of panoramas such as the spherical panoramas (Figs. 15A-15B) and the planar 
panoramas (Figs. 16A-16B). Shum teaches a setting step of zooming/panning the panorama and 
setting the spherical panoramas (column 23-24) by the interactive user interface so that the 
mapping mode changes in which each resulting panorama has different mapping surfaces 
(column 23-24). Shum further teaches determining the constraints governing the mapping mode 
of the mapping onto the projection plane wherein the constraints incorporates the predetermined 
panoramic view of a 3D scene or the predetermined camera orientation matrix by the governing 
linear systems of equations. Shum further teaches generating a warning message to indicate that 
the constraint equations are unsolvable in a case in which the synthesized image is not aligning 
with any portion of the projected model in response to the change in the geometric constraints 
determined by the projection plane or in response to the change in the panoramic view. The 
geometric constraints change due to the change in the projection plane from a planar panorama 
to a spherical panorama wherein the planes within the spherical panorama are changed to 
spherical planes and the panorama view can be changed as well by zooming/panning panorama 
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and/or selecting a portion of the panorama. Therefore, un-solvability of the linear equations 
occurs in the process of changing the mapping mode because the change in the mapping mode or 
the change in the geometric surface also changes the geometric constraints governing the linear 
equations. When the linear systems of equations are not solvable, a user can switch the 
mapping plane orientation or changing the mode of projection surface to meet the 
constraint equations requirements wherein the plane normals have changed. Moreover, the 
projection space can be in the form of any shape such as spherical or planar and by changing the 
camera's point of view and/or the panorama view and the coordinates and directions of the plane 
normals, the projection space is effectively changed and accordingly the mapping mode is 
changed ( column 13 -24 V 

However, Shum teaches changing the mapping mode by zooming and panning the screen 
images of the interior scene shown in the initial panoramic view of Fig. 10A (See Figs. 1 1A and 
1 IB and column 23-24) and changing the type of panoramas such as the spherical panoramas 
(Figs. 15A-15B) and the planar panoramas (Figs. 16A-16B). Shum teaches a setting step of 
zooming/panning the panorama and setting the spherical panoramas (column 23-24) by the 
interactive user interface so that the mapping mode changes wherein each resulting panorama 
has different mapping surfaces in accordance with the user input commands (column 23-24) 
wherein the panorama mapping surfaces change as a result of the zooming and panning. It is 
noted that this user interface can be automated by the computer software. 



Claim Rejections - 35 USC §103 



Application/Control Number: 09/624,385 Page 7 

Art Unit: 2672 

1. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

2. Claims 1 , 5, 8, and 1 1 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Xiong et al. U.S. Patent No. 6,434,265 (hereinafter Xiong), and further in view of Shum et al. 
U.S. Patent No.,6,271,855 (hereinafter Shum) and Teo U.S. Patent No. 6,246,413 (Teo). 

3. Re Claim 1: 

(a) Xiong teaches an image synthesis method comprising: 

An input step, of inputting a plurality of image data representing a plurality of images 
(e.g., column 9, lines 20-30); 

A placement information generating step, of generating placement information about the 
horizontal and vertical placement direction of the plurality of images, determined by the 
horizontal and vertical placement direction of the plurality of images represented by the image 
data inputted in the input step (e.g., column 16, lines 35-45); 

A placement information obtaining step of obtaining the placement information about a 
plurality of images in which adjacent images have a common subject region (e.g., in column 4, 
lines 5-40, Xiong teaches a method for constructing a panorama from rectilinear images in 3D 
through projective registration and calibration including: (1) the projective registrations of 
overlapping images, (2) calibration and global optimization of these images, a self calibration in 
which 2D image planes are positioned as 3D planes in space) ; 
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A setting step of selecting and setting one mapping mode out of a plurality of mapping 
modes, each mapping mode corresponding to a different mapping surface (e.g., in column 8, 
lines 18-58 o/Xiong, it is stated that "overlapping photographs are analyzed to determine what 
orientation the photographs were taken in order to establish a common ground for subsequent 
operations and the panorama is constructed on a particular geometry that will best facilitate the 
rendering of the projection of the panorama onto a chosen viewing plane for viewing". The 
Xiong discloses some typical geometry on which panoramas are formed; In column 8, lines 18- 
58, Xiong further discloses that panorama is constructed on a particular geometry that will best 
facilitate the rendering of the projection onto a chosen viewing plane such as cubic, polyhedral, 
cylindrical and spherical geometries) ', and 

A synthesis step of combining the plurality of images by using the mapping mode set in 
the setting step {e.g., in column 4, lines 5-40 of Xiong, it is stated that the composing or blending 
in which images are ready to be re-projected to a 3D environment map with pixels in overlap 
regions being composed from multiple; In column 8, lines 18-58, Xiong further teaches that 
overlapping photographs are analyzed to determine what orientation the photographs were 
taken in order to establish a common ground for subsequent operations and the panorama is 
constructed on a particular geometry that will best facilitate the rendering of the projection of 
the panorama onto a chosen viewing plane for viewing. The Xiong further discloses some typical 
geometry on which panoramas are formed)', 

A changing step, of changing the mapping mode (In column 4, lines 40-50, it is stated 
that (< the projection module may be controlled through the user interface 230 as well, to allow a 
user to select what geometry will be projected onto ". Therefore, Xiong teaches a changing step 
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through the user interface 230 by selecting a geometric surface out of a plurality of geometric 
surfaces each corresponding to a different mapping surface) . 

- The Examiner interprets "a placement information obtaining step of obtaining placement 
information about a plurality of images in which adjacent images have a common subject 
region" as an automatic registration and calibration step of registering the overlapping 
images and capturing common overlapping areas between overlapping images and 
minimizing the average squared pixel intensity difference with respect to certain 

trans formation parameters . 

- The Examiner interprets the mapping mode as mapping images onto a geometric surface 
such as a planar or a cylindrical surface (Applicant's specification, page 1, lines 20-25). 
Accordingly, the Examiner interprets the setting step of setting a mapping mode as the 
selecting step of selecting a geometric surface. In column 8, lines 18-58, Xiong discloses 
that panorama is constructed on a particular geometry that will best facilitate the 
rendering of the projection onto a chosen viewing plane (a geometric surface) such as 
cubic, polyhedral, cylindrical and spherical geometries. In column 4, lines 40-50, it is 
stated that "the projection module may be controlled through the user interface 230 as 
well, to allow a user to select what geometry will be projected onto". Therefore, Xiong 
teaches a selecting step of selecting a geometric surface out of a plurality of geometric 
surfaces each corresponding to a different mapping surface in accordance with the 
obtained placement information. As applied to the present application, Xiong fulfills the 
claimed limitation of a setting step of setting one mapping mode out of a plurality of 
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mapping modes each corresponding to a different mapping surface in accordance with 
said obtained placement information. 

(b) However, Xiong is silent to "a setting step, of automatically selecting and setting one 
mapping mode out of a plurality of mapping modes, each mapping mode corresponding to a 
different mapping surface, without a user intervening to determine the mapping mode or the 
corresponding mapping surface, in accordance with the horizontal and vertical placement 
direction of the plurality of images having a common subject region obtained in the placement 
information obtaining." 

(c) Teo teaches the claim limitation of a setting step, of automatically selecting and 
setting one mapping mode out of a plurality of mapping modes, each mapping mode 
corresponding to a different mapping surface, without a user intervening to determine the 
mapping mode or the corresponding mapping surface, in accordance with the horizontal and 
vertical placement direction of the plurality of images having a common subject region obtained 
in the placement information obtaining (Teo teaches that the viewing software can automatically 
invert the projection by automatically straightening out the portion of the panoramic image 
being displayed without the user intervention and thereby automatically setting the mapping 
mode by the viewer software in column 7-8. Furthermore, Teo teaches the image processor for 
automatically transforming the first digital panoramic image into a second digital panoramic 
image corresponding to the projection of the scene onto the second surface geometry and 
therefore automatically selecting and setting the mapping mode of mapping the images to the 
selected surface geometry in column 7. Teo teaches in column 10 the mapping mode of mapping 
onto the planar surfaces such as mapping the panorama images onto a cube and the mapping 
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mode of mapping the panorama images onto polyhedral surfaces to form cylindrical panorama 
images. Teo teaches horizontal placement of image by projecting a scene onto a cube so that 
horizontal lines of the scene appear horizontal in the panoramic image and vertical placement of 
image by projecting the panoramic image on the vertical cylinder or by rotating the polyhedral 
surface relative to a cylindrical panoramic image and making adjustments to the sides and 
edges of the polyhedral surface so that the placement of images appear vertically on the 
mapping surface. Because the mapping surface has been adjusted without the user } s action. Teo 
discloses the automatically selecting and setting of the mapping mode of mapping the scene to 
the automatically adjusted surfaces . Although Teo discloses user may intervene in changing a 
mapping surface, e.g., Teo teaches the aspect of automatically selecting and setting the mapping 
mode without a user 's action after a user changes a mapping surface, see column 11. lines 1-9. 
Teo does not teach away from the claim limitation of "without a user intervening to determine 
the mapping mode " because a user changes a mapping surface, not a mapping mode. The 
mapping mode is automatically determined by the viewer software. Moreover, any manual 
procedure of changing a mapping surface can be converted to a software-automated procedure. 
See In re Venner, 120 USPQ 192, 194; 262 F2d 91 (CCPA 1958)). 

(d) It would have been obvious to one of ordinary skill in the art to have incorporated the 
Teo's automatic selecting and setting step because Xiong discloses in column 8, lines 18-58 that 
a panorama is constructed on a particular geometry that will best facilitate the rendering 
of the projection onto a chosen viewing plane (a geometric surface) such as cubic, polyhedral, 
cylindrical and spherical geometries. Xiong describes that a user can alternatively intervene to 
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select a viewing plane. Therefore Xiong at least suggests automatically selecting and setting a 
mapping surface without a user intervention to determine the mapping mode. 

(e) One having the ordinary skill in the art would have been motivated to do this because 
selecting and setting a mapping mode out from a plurality of mapping modes serves for the 
purpose of eliminating visible artifacts and providing a best projection and adjustment to the 
mapping surface can be done by the software-automated procedure (Teo column 10). 

(f) However, Xiong and Teo are silent to issuing warning wherein the warning being 
issued in a case in which the synthesized image exceeds a predetermined angle of view when a 
cylindrical mapping mode is changed to a plane mapping mode in the claim limitation of "a 
generating step, of issuing , when an image formed by changing the mapping mode in the changing 
step does not comply with a predetermined condition set in accordance with the mapping mode, a 
warning and generating a synthesized image in accordance with the predetermined condition, the 
warning being issued in a case in which the synthesized image exceeds a predetermined angle of 
view" (i.e., generation of a warning message). 

(g) Shum implicitly teaches issuing warning wherein the warning being issued in a case 
in which the synthesized image exceeds a predetermined angle of view when a cylindrical 
mapping mode is changed to a plane mapping mode in the claim limitation of "a generating step, 
of issuing , when an image formed by changing the mapping mode in the changing step does not 
comply with a predetermined condition set in accordance with the mapping mode, a warning and 
generating a synthesized image in accordance with the predetermined condition, the warning 
being issued in a case in which the synthesized image exceeds a predetermined angle of view 
while simultaneously generating and displaying the synthesized image within the predetermined 
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range of viewing angle" (e.g., Shum teaches a plurality of mapping modes such as zooming and 
panning screen images of the interior scene shown in the panoramic view of Fig, 10A. See Figs. 
11 A and JIB and column 23-24. Shum also teaches the type of panoramas such as the spherical 
panoramas in Figs. 15A-15B and the planar panoramas in Figs. J6A-J6B. Shum teaches a 
setting step of zooming/panning the panorama and setting the spherical panoramas in column 
23-24 by the interactive user interface so that the mapping mode changes wherein each resulting 
panorama has different mapping surfaces in accordance with the user input commands. Shum 
further teaches determining the constraints governing the mapping mode of the mapping onto the 
projection plane wherein the constraints incorporates the predetermined angle of view such as 
the panoramic view of a 3D scene or the predetermined camera orientation matrix by the 
governing linear systems of equations. Shum further teaches generating a warning message to 
indicate that the constraint equations are unsolvable in a case in which the synthesized image is 
not aligning with any portion of the projected model when the projection plane is changed or 
when the angle of view is changed in response to the user commands, e.g., the projection plane is 
changed from a planar panorama to a spherical panorama wherein the planes within the 
spherical panorama are changed to spherical planes and the panorama view can be changed as 
well by zooming/pann ing panorama and/or selecting a portion of the panorama. When the linear 
systems of equations are not solvable, a user can switch the mapping plane orientation or 
changing the mode of projection surface to meet the constraint equations requirements by 
changing the plane normals. Moreover, the projection space can be in the form of any shape 
such as spherical or planar and by changing the camera 's point of view and/or the panorama 
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view and the coordinates and directions of the plane normals, the projection space is effectively 
changed and thereby the mapping mode is changed. See column 13-24. ) 

(h) It would have been obvious to one of ordinary skill in the art to have incorporated the 
Shum's warning message generation step because Xiong suggests a generating step of generating 
a synthesized image in accordance with the predetermined condition (Xiong column 3, lines 35- 
55; column 17, lines 15-67; column 18, lines 1-4). Moreover, Xiong teaches in figures 2 and 3 a 
user interface and a global optimization that provides feedback to the computer system such as 
issuing warning messages on the computer monitor 218 when the pair-wise objective function is 
not desirable for a poor selection of the projection viewing plane (and the resulting panoramas 
will have imperfectly aligned images that give shadow or ghosting effects) due to a poor 
selection of the projection viewing plane and the resulting panoramas will have imperfectly 
aligned images that give shadow or ghosting effects. Xiong also points to a seamless multi- 
resolution average blending method that would result in an absent of shadow effects (column 14, 
lines 1-45). Xiong further teaches a human interaction being present throughout the image 
synthesis process to provide feedback to the computer system in all the nonlinear optimizations 
to let users monitor the progress of the system and allow them to intervene when necessary and 
therefore the claimed limitation suggests an obvious modification of Xiong because providing 
feedback to users is similar to issuing a message to users in a user interaction with the nonlinear 
optimizations when the changing step does not comply with a predetermined condition set or 
user-selected parameter set. 

(i) One having the ordinary skill in the art would have been motivated to do this because 
this would have provided the user the informative message such as issuing a warning message 



Application/Control Number: 09/624,385 Page 1 5 

Art Unit: 2672 

through the user interaction when convergence to the predetermined condition setting cannot be 
obtained in the image synthesis (Xiong column 17, lines 15-67, column 18, lines 1-4) for non- 
solvable optimization problems (Shum column 18, lines 38-67). 



Claim 5: 

The claim 5 recites all the limitations of claim 1 and adds the limitation of "a displaying 
step of displaying a cuttable rectangular region " Xiong teaches how to align images more 
precisely by changing the coordinates for positioning an image. Xiong further teaches placing the 
images 1210 at selected tangents on the viewing sphere 1220 (figure 12, and column 17, lines 
12-65). 



Claim 8: 

Xiong teaches how to align images more precisely by changing the coordinates for 
positioning an image. Xiong further teaches placing the images 1210 at selected tangents on the 
viewing sphere 1220 (figure 12, and column 17, lines 12-65). 

4. Claim 11: 

The claim 1 1 encompasses the same scope of invention as that of claim 1 except 
additional claimed limitation of a computer-readable storage medium having a program for 
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implementing image synthesis method. However, Xiong further discloses the claimed limitation 
of a computer-readable storage medium having a program for implementing image synthesis 
method (e.g., in column 3, lines 54-57, it is stated "a program residing in system memory 220 
which stores output data and other data"). 



Conclusion 

Applicants amendment necessitated the new ground(s) of rejection presented in this 
Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). 
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1 .136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1 .136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the date of this 
final action. 
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Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jin-Cheng Wang whose telephone number is (571) 272-7665. 
The examiner can normally be reached on 8:00 - 6:30 (Mon-Thu). 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Mike Razavi can be reached on (571) 272-7664. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should. you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 
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